Method and apparatus for creating non-linear motion picture transitions

ABSTRACT

Creating a transition between a first sequence of video frames and a second sequence of video frames. The method includes storing a table of values that express a non-linear response to certain levels of light, generating a transition between the sequences, and adjusting the intensity at which material of the first sequence is displayed relative to the intensity at which material of the second sequence is displayed within the transition, based on the values stored in the step of storing.

RELATED APPLICATIONS

This application is a continuing application of, and claims the benefitunder 35 U.S.C. §120 to:

U.S. Ser. No. 08/932,557, filed Sep. 19, 1997 now U.S. Pat. No.6,532,043, which is a continuation of U.S. Ser. No. 08/665,277, filedJun. 17, 1996, now U.S. Pat. No. 5,812,216, which is a continuation ofU.S. Ser. No. 08/230,050, filed Apr. 19, 1994, now U.S. Pat. No.5,528,310, which is a continuation in part of U.S. Ser. No. 08/049,028,filed Apr. 16, 1993, now issued as U.S. Pat. No. 5,440,348.

FIELD OF INVENTION

The invention relates generally to systems and methods for editing andcomposing motion picture material, and more particularly to operationson transitions between scenes in such material.

BACKGROUND OF THE INVENTION

In assembling individual motion picture (i.e., film or video) scenesinto larger compositions, the scenes are linked together by transitions.Most transitions are simple cuts, but a significant number are otherforms of gradual transitions. These include dissolves (or cross-fades),wipes, and various specialized transitions such as blends rotations andpeels.

In a dissolve, a first scene is gradually dimmed while a second scene isgradually brightened, so that the viewer sees the transition as acontinuous shift from the first scene to the second scene. During all ofthe types of gradual transition effect, as the term is to be understoodin the context of this application, the first scene and second are bothvisible to some extent at the same time. For example, in a wipe, part ofthe second scene is displayed in a portion of the first scene, and thisportion is gradually made larger until the second scene occupies theentire display area.

Preparing gradual transitions using 35mm motion picture film isgenerally performed by specialized laboratories. 16mm transitions areprepared using a technique called A-B rolling, in which two prints ofthe film are prepared with alternating scenes and then combined to yieldthe gradual transitions. Both of these techniques usually require thefilm to be conveyed to a processing laboratory in order to generate thetransition in final form.

Video editing and composition systems also allow for the creation oftransitions. In these systems, it is known to specify a gradualtransition by its centerpoint. By centerpoint is meant the point in timewithin the transition half way between the beginning and end of thetransition. The editor manipulates the transition much like a cut, butupon playback, the transition begins before the centerpoint, and endsafter the centerpoint. The user may also manually generate thetransition using an effects bar control, which is a control lever thatoften resembles a joystick. This allows the user to control the rate atwhich the transition takes place. These manually-generated transitionprofiles may also be stored.

It is also known to display a series of independent monitors with thematerial from the first scene above a second series of independentmonitors with the material from the second scene. In the top row, eachmonitor shows a successive frame of the first scene, and in the bottomrow, each monitor shows a corresponding successive frame of the secondscene. This system may allow the user to view several frames from eachscene in a transition simultaneously. The Montage Picture Processorsystem, available from Montage R&D Corp. of Littleton, Mass., uses thisapproach.

SUMMARY OF THE INVENTION

In general, the invention pertains to a transition editing method forcreating a transition between a first sequence of video frames and asecond sequence of video frames. The method includes storing a table ofvalues that express a non-linear response to certain levels of light,generating a transition between the sequences, and adjusting theintensity at which material of the first sequence is displayed relativeto the intensity at which material of the second sequence is displayedwithin the transition, based on the values stored in the step ofstoring.

In another general aspect, the invention pertains to the editing oftransitions between sequences of video frames. This includessimultaneously displaying the frames in the first and second sequencesat the beginning of the transition, and the frames in the first andsecond sequences at the end of the transition. The position of two ofthe four frames may then be altered in their respective sequence toredefine the transition in response to move commands. The rate of changewithin the transition may also correspond to a predefined filmcharacteristic.

The transition editing method of the invention allows for efficient,versatile and precise editing of motion picture source material. Becausethe first and last frames of the two scenes in the transition are bothshown, it is easy for the user to see material that is not intended tobe part of the transition. Such material may include booms, slates,microphones and the like. These unintended events might otherwise behard to see, especially at the edges of the transitions where one of thescenes may be quite dim. The controls and displays of the invention arealso meaningfully and efficiently laid out. The film-matching transitionfeature of the invention allows an editor to edit video tape usingtransitions that are similar to film transitions, to achieve a morefilm-like effect on a video copy. This enables the user to accuratelypreview material on the computer that is ultimately to be printed onfilm, without the cost and delay of laboratory work. Alternatively, theuser can experiment with different types of transition characteristics,to achieve desired aesthetic effects.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the present invention, together with otherand further objects, advantages and capabilities thereof, reference ismade to the accompanying drawings, which are incorporated herein byreference and in which:

FIG. 1 is a block diagram of a motion picture composing system accordingto the invention;

FIG. 2 is a drawing of an exemplary interface screen for the system ofFIG. 1;

FIG. 3 is a schematic representation of a transition;

FIG. 4 is an alternative embodiment of display portions of the screen ofFIG. 2;

FIG. 5 is a sketch of a response curve for photographic film plottingdensity of pigmentation against level of illumination;

FIG. 6 is a schematic diagram of a transition using the film of FIG. 5;and

FIG. 7 is a flow diagram illustrating the generation and display of anon-linear transition.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring to FIG. 1, an exemplary motion picture composition system 10according to the invention includes a transition display module 12, aplayer module 14, and a timeline display module 16. A data path 11 linksa storage unit 18 with these modules. This data path may be broken downinto several data lines 13, 15, 17. These data lines may be dedicated todifferent synchronized media channels, such as video, audio, andcomposition channels. These lines need not be separate physical lines.An input data path relays user input from a user input device 20, suchas a pointing device, to the transition display module. The transitiondisplay module 12 relays transition display window information totransition display windows 32, 34, 36, 38 (see FIG. 2). The transitiondisplay module also relays transition information 21 to the playermodule 14, which will enable the player module to play back the entiretransition on a transition playback display window 40. The timelinedisplay module generates timeline information to be displayed on thedisplay as a pair of timelines 42, 44. The timeline display module alsoprovides information about user-initiated changes of transitions to thetransition display module.

A system according to the invention may be developed using a variety oftypes of computer systems. In one embodiment, an Apple MacIntosh and the“C” computer language were used, but naturally the invention should notbe read to be limited to this particular platform. The transitiondisplay module 12, player module 14 and timeline display module 16 maybe implemented in software and/or hardware and may or may not becontained within the computer system. A software package entitled “AvidMedia Composer”, available from Avid Technology, Inc. of Tewksbury,Mass., includes some of these features. The storage may include ahigh-speed, high-capacity internal magnetic hard disk drive. For furtherinformation on computer-based media storage and playback of video andaudio information, see Peters et al., “VIDEO AND AUDIO TRANSMISSIONSYSTEM AND METHOD”, U.S. Pat. No. 5,045,940, issued Sep. 3, 1991, hereinincorporated by reference.

Referring to FIG. 2, the display interface 30 presented by thecomposition system to the user includes a series of four transitiondefinition display windows 32, 34, 36, 38. The interface also includes atransition playback display window 40, a coarse timeline 42, a finetimeline 44, a command bar 46, and a mode bar 48.

Referring to FIGS. 2 and 3, the transition between a first scene “A” 50and a second scene “B” 52 begins where a first frame position 54 startsto include material from both of the scenes. In the case of a dissolve,this first frame position will show a frame 56 from the first scene atthe same time as a frame 58 from the second scene. In this first frameposition, the frame from the first scene will be displayed much morebrightly than the frame from the second scene. As the dissolvecontinues, frames from the first scene will become dimmer as frames fromthe second scene become brighter. At a final frame position, a lastframe 62 from the first scene will be displayed much more dimly than thecorresponding frame 64 from the second scene. This type of transition issymbolized by a diagonal line 57 representing this gradual shift.

The transition definition display windows each correspond to framepositions either at the beginning 54 or the end 60 of the transition.The first window 32 of FIG. 2 displays the first frame 56 from the firstscene in the transition. The second window 34 displays the last frame 62from the first scene in the transition. The third window 36 displays thefirst frame 58 from the second scene in the transition. The fourthwindow displays the last frame 64 from the second scene in thetransition.

In operation, a user of the composing system of the invention will firstmanipulate the coarse timeline 42 to determine the rough area in whichthe transition is to be added. The user may then manipulate the finetimeline 44 to define a precise position for the edit. Using a mouse orother computer input device, the user may select two scenes to bedisplayed and place them in a composing area 45 of the timeline. To thencreate a gradual transition, the user may select one of a series oftransition types and drag it to a position between the two scenes on thecomposition track 45 of the timeline. The system thereafter displays thetransition as a transition control box 49.

The user may then drag the transition start control line 70 to move thestart of the transition earlier or later in time. Similarly, the usermay move the end transition control line 72 to end the transitionearlier or later in time. In moving the transition start line 70, thefirst transition display window 32 will display an earlier frame of thefirst scene. The third transition window 36 will also display an earlierframe of the second sequence. Moving the transition end control line 72similarly affects the second 36 and fourth 38 transition displaywindows.

The user may also move the central transition position 53 of the firstscene. This may be done by manipulating the central transition commandline 77 on the fine timeline. When the user moves the center of thetransition for the first scene in this way, the first and secondtransition display windows 32, 34 are updated correspondingly, but thethird and fourth transition display windows do not change. This allowsthe user to start the transition at a different time within the firstscene.

Similarly, the user may move the center control for the second scene.This may be done with the central transition command line 77 modified bya mode control. In response to this command, the third transitiondisplay window 36 and the fourth transition display window 38 will beupdated, leaving the first and second transition display windows 32, 34unchanged. This allows the user to chance the point in the second sceneat which the transition begins. The user may also move both transitioncenters at the same time, in a similar manner. The user may play thewhole transition on the playback display window at any time, undercontrol of the command bar 46.

Referring to FIG. 4, an alternative embodiment of the invention alsoincludes four transition display windows and a playback window. In thisembodiment, however, a series of four button bars 80, 82, 84, 86 areprovided in a pattern surrounding the display windows. A button bar isan area of the screen containing “buttons” or control areas, whichrespond to user input from a pointing device.

The first button bar 80 is placed above the first and second transitiondisplay windows and centered on a line roughly between them. The secondbutton bar 82 is placed below the third and fourth transition displaywindows and centered on a line roughly between them. The third buttonbar is placed to the left of the first and third transition displaywindows and centered on a line roughly between them. The fourth buttonbar is placed to the right of the second and fourth transition displaywindows and centered on a line roughly between them. This arrangementand the corresponding functionality allow for efficient editing and areeasy to understand.

The button bars may each include a series of control icons or buttons.In one embodiment, each button bar includes four buttons 90, 92, 94, 96.A first 90 of these buttons is a single frame advance button, whichpermits the user to step through the motion picture material to bedisplayed one frame at a time. A second button 92 is a multiple frameadvance button, which allows the user to advance through the material ata rate of approximately 0.3 seconds per actuation. Conversely, a singleframe reverse button 94 and multiple frame reverse button 96 are alsoprovided. Other controls may also be provided on the button bars, asdiscussed later in the present specification.

The first button bar 80 controls the centerpoint of the transition forthe first scene, and the second button bar 82 controls the centerpointof the transition for the second scene. The third button bar 84 controlsthe start of the transition for both scenes, and the fourth button bar86 controls the end of the transition for both scenes. By clicking onthe button bars, the user may therefore quickly adjust the parameters ofthe transition.

Other control arrangements are possible. For example, one button barmight be provided for each of the transition display windows, allowingthe user to manipulate the position of each frame position individually.Another possible approach is to provide a second display window, similarto the display window 40. These two display windows can be used toindividually, but simultaneously, play back the portions of the twoscenes in the transition. They can be placed one above the other betweenthe first and second, and the third and fourth transition displaywindows.

According to another aspect of the invention, the composition system cancompensate for non-linearities in film response. Referring to FIG. 5,the density of pigmentation 101 of film resulting from a given exposurelevel 103 varies in a non-linear manner. The response curve for such afilm generally includes what are known in the art as a toe portion 100and a shoulder portion 102 in its deviation from an ideal response 104.Curves for specific film types are usually available from motion picturefilm manufacturers.

In the generation of a transition, the ends of the scene to be used inthe transition are generally printed by exposing them to linearlyincreasing levels of light. Because the film responds to light in anon-linear way, the resulting transition 106 will also be non-linear, asillustrated in FIG. 6. These transitions start out with a low rate ofchange from the first scene to the second scene. The rate of change thenincreases, and finally tapers off towards the end of the transition.This effect is generally absent in electronically generated videotransitions.

The non-linear transition characteristics of film, however, can be morepleasing to the eye. Furthermore, it is often advantageous to performthe bulk of the editing of a piece using a computerized compositionsystem and then transfer the edits to a final film copy. It is thereforeuseful to provide transitions in the composition system that willsimulate the ultimate effect on film.

Referring to FIGS. 1 and 2, the storage 18 of the composition system 10may hold a series of response curve data tables for various types offilm. When using the system, the user may select one of these tables fora desired film type to adjust the parameters of the transition to beplayed back on the display interface 30. The system then retrievesvalues from the selected table and determines from them the relativeintensities for concurrently displaying frames from the first and secondscenes during the transition. For example, in the beginning of thetransition of FIG. 6, frames from the first scene will be displayed morebrightly relative to the first scene than they would in a lineartransition. This is because the intensity of the first frames of thefirst scene are being affected by the table values corresponding to theshoulder portion 102 of the selected film's response, and the intensityof the first frames of the second scene are being affected by the tablevalues corresponding to the toe portion 100 (see FIG. 5).

The non-linear characteristics of a film type can be applied todisplaying a transition in different ways. In one approach, thenon-linear film characteristics are applied to the individual pixels inthe two scenes in the transition. In an alternative approach, thenon-linear film characteristic is applied to the frame as a whole.

In the first approach, referring to FIGS. 1 and 7, once the playermodule 14 has received transition information 21 from the transitiondisplay module 12, it creates and displays the transition in thefollowing manner. The player module begins by retrieving, from a pixelsequence 116 for the first scene, the first pixel for the first framewhich is to appear in the transition. The player module also retrieves,from a pixel sequence 118 for the second scene, the first pixel for thefirst frame which is to appear in the transition.

The player module then prefades each of these pixels (steps 112, 114),to obtain first and second prefaded pixel values 120, 122. The prefadingoperations adjust the relative intensity of the pixels in the twoframes, by multiplying them by respective complementary floating pointblending factors. The player module then combines the two prefadedvalues to form an address, and uses that address to access a storedtable (step 124). The entry in the table accessed by that address is acomposite pixel value 126, which can be displayed on the display 40(step 128).

The player module then repeats these operations for the remaining pixelsin the first frames of the first and second scenes. The result is that ablended first frame is displayed. In an ordinary dissolve, the blendingfactors each stay the same for all pixels of each blended frame.

These operations are then again applied successively to the pixels inthe remaining frames in the transition. In an ordinary dissolve, thefirst blending factor decreases linearly as the transition progresses,and the second blending factor increases linearly as time progresses.This causes the first scene to fade out as the second scene fades in.

These blending factors add up to a value of one throughout thetransition. For example, at a point in the transition that calls for thefirst scene to be displayed twice as brightly as the second frame, theblending factor for the first scene will be ⅔, and the blending factorfor the second scene will be ⅓. The prefaded pixel value 120 for thepixel in the first frame would therefore be reduced in intensity by ⅓,and the prefaded pixel value 122 for the pixel in the second frame wouldbe reduced in intensity by ⅔. It is noted that if the prefaded valueswere to be added for the whole transition, instead of being transformedby the table, the result would be a linear video fade.

The table contains response curve data for a particular type of film,which allows the transition to be displayed as a non-linear transition.Each entry in the table holds a precomputed composite pixel valueaccessed by an address made up of both the prefaded pixel values 120,122. This precomputed value corresponds to the sum of these two pixelsafter their intensities have each been adjusted according to theircorresponding positions on the film characteristic. It is noted that thetable can be precomputed for two different film characteristics, one forthe first scene, and one for the second. One of those characteristicscould be linear.

This approach to displaying a transition results in portions ofindividual frames fading in and fading out at different rates, dependingon their intensity level and the chosen film characteristic curve. Thisquality of a transition is visibly discernable by the user, and mayproduce desirable aesthetic effects.

Consider, for example, fading away from a first scene which includes theface of a black cat with bright green eyes and sitting against a darkbackground. With the film characteristic shown in FIG. 5, the eyes ofthe cat will tend to persist in the displayed material after the outlineof the cat's face has disappeared. This is because the brighter portionsof the frame tend to be affected by the knee 102 of the characteristicuntil later in the transition, and the darker portions of the frame areaffected by the toe 100 of the characteristic earlier in the transition.

The motion picture composing system according the invention can alsostore arbitrary or user-specified non-linear response curves. Theseresponse curves may provide certain desired effects to a transition. Onepossible arbitrary characteristic would be a characteristic similar tothat of FIG. 5, but mirrored about the ideal response 104. In fadingaway from a scene, with this characteristic, the dark portions of thescene will tend to persist longer than the lighter portions of thescene. In fading away from the cat's face, for example, the silhouetteof the cat would tend to persist as the eyes of the cat grow dimmer. Byallowing users to experiment with these characteristics, they can obtaina variety of different aesthetic effects for the transition.

The second approach to displaying non-linear transitions differs fromthe first in that the prefading operations 112, 114 are not carried outlinearly over time, and the look-up operation 124 is replaced with asumming operation. In this type of transition, the prefading operationsfollow a representative film characteristic, such that all of theintensities in the same frames fade in or out at the same rate. Thisrepresentative film characteristic can, for example, track thetransition that a pixel of 50% intensity would follow using the firstapproach.

The second approach is not an accurate reproduction of the mechanismtraditionally used to create film transitions, but produces a transitionthat resembles a film transition more closely than does a linear videotransition. In particular, this type of transition will follow thegeneral cadence of a film transition, but it will not have the selectivefading characteristics described above. Generating this type oftransition generally requires fewer computational resources, however,than the type of transition generated using the first approach.

It is common to employ compressed video information in composingdigitally stored video images. For example, JPEG, MPEG, and otherdiscrete cosine transform (DCT) compression methods can be used. Thesegenerally end with a multiplication step, and this multiplication stepcan be modified to perform the prefading operations as well.

For example, in a system employing the JPEG compression standard, afinal operation is a conversion between YUV 4:2:2 and RGB formats, andthis conversion is generally performed with a 3×3 vector matrixmultiplication operation. In this type of system, the elements in thetransformation matrix can all be multiplied by the blending factor, suchthat only a single multiplication operation is required to both convertand prefade. The multiplication can take place in a dedicated JPEGcompression processor, with the table look-up or addition steps beingperformed by a host system. This results in efficient generation of thetransitions for both the first and second approaches.

Alternatively, the transitions can be partially or fully computed by adedicated video processor. This can be done either before or afterconverting from YUV 4:2:2 to RGB representations, but YUV 4:2:2representations can be more compact and may require fewer operations.Since such processors generally have reduced amounts of memory, it maybe useful to fold the film characteristic table (if it issymmetrical—i.e., one film characteristic is used for both scenes), orotherwise compress it. One type of processor which can be used toperform this sort of processing is described in a copending PCTapplication designating the United States, entitled: “Meda Pipeline withMultichannel Video Processing and Playback”, by Jeffrey Kurtze et al.,assigned to Avid Technology, Inc., filed on Apr. 18, 1994, and which isherein incorporated by reference.

It is noted that in the traditional course of generating filmtransitions, the characteristics of two films are involved. The firstfilm is a positive-type film, which is used to film the scenes, and thesecond is a negative-type film which is used in the transition printingprocess. It is therefore desirable to develop composite transitioncharacteristic tables to take both films into account. These compositecurves can be derived by convolving the two individual filmcharacteristic curves.

A system according to the invention may also permit motion in thetransition display windows. This allows editors to define these pointsin a dynamic fashion. For example, certain moving scenes are easier toedit while the editor is perceiving the tempo of the scene. In anembodiment permitting these types of dynamic transition editingfunctions, further buttons may be provided on the button bars, such asbuttons having play, fast forward, rewind, jog and shuttle functions. Itis noted that the four transition display windows need not display thefirst and last frames of the transition, but instead may display theframes immediately preceding and immediately following the transition.

The system according to the invention also permits versatile editingoperations on the soundtrack of motion picture material. Although thistype of transition only affects audio information, it can be specifiedeasily using the transition display windows. For example, the user mightwant to create the effect of a door opening with a train outside of thedoor. With the system of the invention, the user may create this effectby defining a transition on one or more of the audio tracks of thematerial from background room sounds to train noise. This transition maybe defined to start at the first frame in which the door starts to open,which is easily found using the transition display windows.

While there have been shown and described what are at present consideredthe preferred embodiments of the present invention, it will be obviousto those skilled in the art that various changes and modifications maybe made therein without departing from the scope of the invention asdefined by the appended claims.

What is claimed is:
 1. A computer implemented method rendering of atransition from a first scene to a second scene so as to simulate thetransition as if it were made on film, comprising: determining, for eachimage in the transition, a blend factor that indicates how acorresponding image from the first scene and a corresponding image fromthe second scene are combined to create the image in the transition;combining, for each image in the transition, pixels in the image fromthe first scene with pixels in the image from the second scene accordingto both the blend factor for the image and one or more valuesrepresenting a nonlinear response of film with respect to light, toproduce pixels in the image in the transition.
 2. The computerimplemented method of claim 1, wherein combining comprises: for eachpixel in the corresponding image from the first scene, prefading thepixel according to the blend factor; for each pixel in the correspondingimage from the second scene, prefading the pixel according to the blendfactor; and producing an output value for the pixel in the image in thetransition according to the prefaded pixels from the correspondingimages in the first and second scenes and the values representing thenonlinear response of film with respect to light.
 3. The computerimplemented method of claim 2, wherein producing the output valuecomprises: storing a table representing the output value correspondingto the nonlinear response of film to light corresponding to acombination of the prefaded pixels from the corresponding images in thefirst and second scenes.
 4. The computer implemented method of claim 1,wherein combining comprises: prefading the corresponding image of thefirst scene and the corresponding image of the second scene according toa nonlinear response of film to light of average intensity adjusted bythe blend factor.
 5. A computer program product, comprising: a computerreadable medium; computer program instructions stored on the computerreadable medium that, when executed by a computer, instruct the computerto perform a method for rendering of a transition from a first scene toa second scene so as to simulate the transition as if it were made onfilm, comprising: determining, for each image in the transition, a blendfactor that indicates how a corresponding image from the first scene anda corresponding image from the second scene are combined to create theimage in the transition; combining, for each image in the transition,pixels in the image from the first scene with pixels in the image fromthe second scene according to both the blend factor for the image andone or more values representing a nonlinear response of film withrespect to light, to produce pixels in the image in the transition. 6.The computer program product of claim 5, wherein combining comprises:for each pixel in the corresponding image from the first scene,prefading the pixel according to the blend factor; for each pixel in thecorresponding image from the second scene, prefading the pixel accordingto the blend factor; and producing an output value for the pixel in theimage in the transition according to the prefaded pixels from thecorresponding images in the first and second scenes and the valuesrepresenting the nonlinear response of film with respect to light. 7.The computer program product of claim 6, wherein producing the outputvalue comprises: storing a table representing the output valuecorresponding to the nonlinear response of film to light correspondingto a combination of the prefaded pixels from the corresponding images inthe first and second scenes.
 8. The computer program product of claim 5,wherein combining comprises: prefading the corresponding image of thefirst scene and the corresponding image of the second scene according toa nonlinear response of film to light of average intensity adjusted bythe blend factor.